{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sugarforever/LangChain-Advanced/blob/main/Integrations/AutoGen/autogen_langchain_uniswap_ai_agent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39_S0tWcB-oh"
      },
      "source": [
        "# AutoGen + LangChain + PlayHT Use Case - Super AI Agent that Speaks\n",
        "\n",
        "**`AutoGen`** is a versatile framework that facilitates the creation of LLM applications by employing multiple agents capable of interacting with one another to tackle tasks.\n",
        "\n",
        "**`LangChain`** is an open-source framework designed for software developers engaged in AI and ML. It enables them to seamlessly integrate LLM with external components, facilitating the creation of LLM-driven applications.\n",
        "\n",
        "**`PlayHT`** is a company serving the generative text to speech service.\n",
        "\n",
        "Integrating them together, we are able to build a super cool AI agent that,\n",
        "\n",
        "1. is knowledgeable in certain area\n",
        "2. can **SPEAK**\n",
        "\n",
        "This is the enhanced version of the AI Agent introduced in previous tutorial. We will build the audio feature on top of it. To learn more about it before starting this tutorial, please visit the following link:\n",
        "\n",
        "[AutoGen + LangChain Use Case - Uniswap Protocol AI Agent](https://github.com/sugarforever/LangChain-Advanced/blob/main/Integrations/AutoGen/autogen_langchain_uniswap_ai_agent.ipynb)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NtvjgnBZZjUL"
      },
      "source": [
        "## Use Case - Uniswap Protocol AI Agent that Speaks\n",
        "\n",
        "`Uniswap` is a decentralized exchange that allows users to trade Ethereum-based tokens.\n",
        "\n",
        "In previous tutorial, we already built an AI Agent that can execute tasks require Uniswap protocol knowledge.\n",
        "\n",
        "In this tutorial, let's make the agents answer in **audio**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z73o7bmtb5LH"
      },
      "source": [
        "### Environment Preparation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
          "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
          "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
          "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
        },
        "id": "1VRZnGGGgkhl"
      },
      "outputs": [],
      "source": [
        "%pip install pyautogen~=0.1.0 docker langchain openai tiktoken chromadb pypdf simpleaudio numpy -q"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from dotenv import load_dotenv\n",
        "load_dotenv()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HZ7w_A3nXU8-"
      },
      "outputs": [],
      "source": [
        "import autogen\n",
        "\n",
        "config_list = autogen.config_list_from_json(\n",
        "    \"OAI_CONFIG_LIST\",\n",
        "    filter_dict={\n",
        "        \"model\": [\"gpt-4\"],\n",
        "    },\n",
        ")\n",
        "#\n",
        "# Sample content of OAI_CONFIG_LIST file below:\n",
        "#\n",
        "# [\n",
        "#   {\n",
        "#     \"model\": \"gpt-4\",\n",
        "#     \"api_key\": \"your openai api key\"\n",
        "#   }\n",
        "# ]\n",
        "#"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1WVvbWIDSWMR"
      },
      "outputs": [],
      "source": [
        "from langchain.vectorstores import Chroma\n",
        "from langchain.embeddings import OpenAIEmbeddings\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "from langchain.document_loaders import PyPDFLoader\n",
        "from langchain.memory import ConversationBufferMemory\n",
        "from langchain.llms import OpenAI\n",
        "from langchain.chains import ConversationalRetrievalChain"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BA48TH6Hc_3c"
      },
      "source": [
        "### Steps"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rCrCnRC7cdC-"
      },
      "source": [
        "#### 1. Build up a vector store with Uniswap V3 whitepaper."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LcC3gnqXSZHs"
      },
      "outputs": [],
      "source": [
        "docs = PyPDFLoader('./uniswap_v3.pdf').load()\n",
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000)\n",
        "docs = text_splitter.split_documents(docs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8VoyrUwTShHw"
      },
      "outputs": [],
      "source": [
        "vectorstore = Chroma(\n",
        "    collection_name=\"full_documents\",\n",
        "    embedding_function=OpenAIEmbeddings()\n",
        ")\n",
        "vectorstore.add_documents(docs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PxFsXiHVciOo"
      },
      "source": [
        "#### 2. Set up a conversational retrieval QA chain by LangChain, based on the vector store."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6eRvVjJITKfR"
      },
      "outputs": [],
      "source": [
        "qa = ConversationalRetrievalChain.from_llm(\n",
        "    OpenAI(temperature=0),\n",
        "    vectorstore.as_retriever(),\n",
        "    memory=ConversationBufferMemory(memory_key=\"chat_history\", return_messages=True)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "sXST-2kRTUOs"
      },
      "outputs": [],
      "source": [
        "result = qa(({\"question\": \"What is uniswap?\"}))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "id": "WVi6rT78Tsu_",
        "outputId": "d1808979-3d18-4847-c45a-0f3864bc8819"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "' Uniswap is a noncustodial automated market maker implemented for the Ethereum Virtual Machine.'"
            ]
          },
          "execution_count": 30,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "result['answer']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O7VPXVI_coX4"
      },
      "source": [
        "#### 3. Define a function `answer_uniswap_question`\n",
        "\n",
        "It takes a parameter `question`, calls the QA chain, and answer it by returning the answer from the chain response."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yPThjcdFT1Lw"
      },
      "outputs": [],
      "source": [
        "def answer_uniswap_question(question):\n",
        "  response = qa({\"question\": question})\n",
        "  return response[\"answer\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### 4. Define a function convert_text_to_audio"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from typing import Generator, Iterable\n",
        "\n",
        "import time\n",
        "import threading\n",
        "import os\n",
        "import re\n",
        "import numpy as np\n",
        "import simpleaudio as sa\n",
        "\n",
        "from pyht.client import Client, TTSOptions\n",
        "from pyht.protos import api_pb2\n",
        "\n",
        "def play_audio(data: Generator[bytes, None, None] | Iterable[bytes]):\n",
        "    buff_size = 10485760\n",
        "    ptr = 0\n",
        "    start_time = time.time()\n",
        "    buffer = np.empty(buff_size, np.float16)\n",
        "    audio = None\n",
        "    for i, chunk in enumerate(data):\n",
        "        if i == 0:\n",
        "            start_time = time.time()\n",
        "            continue  # Drop the first response, we don't want a header.\n",
        "        elif i == 1:\n",
        "            print(\"First audio byte received in:\", time.time() - start_time)\n",
        "        for sample in np.frombuffer(chunk, np.float16):\n",
        "            buffer[ptr] = sample\n",
        "            ptr += 1\n",
        "        if i == 5:\n",
        "            # Give a 4 sample worth of breathing room before starting\n",
        "            # playback\n",
        "            audio = sa.play_buffer(buffer, 1, 2, 24000)\n",
        "    approx_run_time = ptr / 24_000\n",
        "    time.sleep(max(approx_run_time - time.time() + start_time, 0))\n",
        "    if audio is not None:\n",
        "        audio.stop()\n",
        "\n",
        "\n",
        "def convert_text_to_audio(\n",
        "    text: str\n",
        "):\n",
        "    text_partitions = re.split(r'[,.]', text)\n",
        "\n",
        "    # Setup the client\n",
        "    client = Client(os.environ['PLAY_HT_USER_ID'], os.environ['PLAY_HT_API_KEY'])\n",
        "\n",
        "    # Set the speech options\n",
        "    voice = \"s3://voice-cloning-zero-shot/d9ff78ba-d016-47f6-b0ef-dd630f59414e/female-cs/manifest.json\"\n",
        "    options = TTSOptions(voice=voice, format=api_pb2.FORMAT_WAV, quality=\"faster\")\n",
        "\n",
        "    # Get the streams\n",
        "    in_stream, out_stream = client.get_stream_pair(options)\n",
        "\n",
        "    # Start a player thread.\n",
        "    audio_thread = threading.Thread(None, play_audio, args=(out_stream,))\n",
        "    audio_thread.start()\n",
        "\n",
        "    # Send some text, play some audio.\n",
        "    for t in text_partitions:\n",
        "        in_stream(t)\n",
        "    in_stream.done()\n",
        "\n",
        "    # cleanup\n",
        "    audio_thread.join()\n",
        "    out_stream.close()\n",
        "\n",
        "    # Cleanup.\n",
        "    client.close()\n",
        "    return 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "convert_text_to_audio(\"Welcome to the Uniswap V3 whitepaper.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wu7gjAv-c4uP"
      },
      "source": [
        "#### 5. Set up AutoGen agents with text-to-audio conversion function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZsXuHf1fgkhl"
      },
      "outputs": [],
      "source": [
        "llm_config={\n",
        "    \"request_timeout\": 600,\n",
        "    \"seed\": 42,\n",
        "    \"config_list\": config_list,\n",
        "    \"temperature\": 0,\n",
        "    \"functions\": [\n",
        "        {\n",
        "            \"name\": \"answer_uniswap_question\",\n",
        "            \"description\": \"Answer any Uniswap related questions\",\n",
        "            \"parameters\": {\n",
        "                \"type\": \"object\",\n",
        "                \"properties\": {\n",
        "                    \"question\": {\n",
        "                        \"type\": \"string\",\n",
        "                        \"description\": \"The question to ask in relation to Uniswap protocol\",\n",
        "                    }\n",
        "                },\n",
        "                \"required\": [\"question\"],\n",
        "            },\n",
        "        },\n",
        "        {\n",
        "            \"name\": \"convert_text_to_audio\",\n",
        "            \"description\": \"Convert text to audio and speak it out loud\",\n",
        "            \"parameters\": {\n",
        "                \"type\": \"object\",\n",
        "                \"properties\": {\n",
        "                    \"text\": {\n",
        "                        \"type\": \"string\",\n",
        "                        \"description\": \"The text to be converted and spoken out loud\",\n",
        "                    }\n",
        "                },\n",
        "                \"required\": [\"text\"],\n",
        "            },\n",
        "        }\n",
        "    ],\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eg9g65hOgkhm"
      },
      "outputs": [],
      "source": [
        "# create an AssistantAgent instance named \"assistant\"\n",
        "assistant = autogen.AssistantAgent(\n",
        "    name=\"assistant\",\n",
        "    llm_config=llm_config,\n",
        ")\n",
        "# create a UserProxyAgent instance named \"user_proxy\"\n",
        "user_proxy = autogen.UserProxyAgent(\n",
        "    name=\"user_proxy\",\n",
        "    human_input_mode=\"NEVER\",\n",
        "    max_consecutive_auto_reply=10,\n",
        "    code_execution_config={\"work_dir\": \".\"},\n",
        "    llm_config=llm_config,\n",
        "    system_message=\"\"\"Reply TERMINATE if the task has been solved at full satisfaction.\n",
        "Otherwise, reply CONTINUE, or the reason why the task is not solved yet.\"\"\",\n",
        "    function_map={\n",
        "        \"answer_uniswap_question\": answer_uniswap_question,\n",
        "        \"convert_text_to_audio\": convert_text_to_audio\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "37cRtpqLdLSZ"
      },
      "source": [
        "### It's time to let the agents SPEAK.\n",
        "\n",
        "Now, let's user the user agent to ask the agents to write an introduction blog for `Uniswap` protocol v3, and **speak it out loudly**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aCdAqig3gkhn",
        "outputId": "0deb286b-a1e4-4b56-e8c4-4ab7ea33173f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
            "\n",
            "\n",
            "I'm writing a blog to introduce the version 3 of Uniswap protocol. \n",
            "Find the answers to the 2 questions below, write an introduction based on them and speak it out loudly.\n",
            "\n",
            "1. What is Uniswap?\n",
            "2. What are the main changes in Uniswap version 3?\n",
            "\n",
            "Start the work now.\n",
            "\n",
            "\n",
            "--------------------------------------------------------------------------------\n",
            "\u001b[33massistant\u001b[0m (to user_proxy):\n",
            "\n",
            "\u001b[32m***** Suggested function Call: answer_uniswap_question *****\u001b[0m\n",
            "Arguments: \n",
            "{\n",
            "  \"question\": \"What is Uniswap?\"\n",
            "}\n",
            "\u001b[32m************************************************************\u001b[0m\n",
            "\n",
            "--------------------------------------------------------------------------------\n",
            "\u001b[35m\n",
            ">>>>>>>> EXECUTING FUNCTION answer_uniswap_question...\u001b[0m\n",
            "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
            "\n",
            "\u001b[32m***** Response from calling function \"answer_uniswap_question\" *****\u001b[0m\n",
            " Uniswap is a noncustodial automated market maker implemented for the Ethereum Virtual Machine.\n",
            "\u001b[32m********************************************************************\u001b[0m\n",
            "\n",
            "--------------------------------------------------------------------------------\n",
            "\u001b[33massistant\u001b[0m (to user_proxy):\n",
            "\n",
            "\u001b[32m***** Suggested function Call: answer_uniswap_question *****\u001b[0m\n",
            "Arguments: \n",
            "{\n",
            "  \"question\": \"What are the main changes in Uniswap version 3?\"\n",
            "}\n",
            "\u001b[32m************************************************************\u001b[0m\n",
            "\n",
            "--------------------------------------------------------------------------------\n",
            "\u001b[35m\n",
            ">>>>>>>> EXECUTING FUNCTION answer_uniswap_question...\u001b[0m\n",
            "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
            "\n",
            "\u001b[32m***** Response from calling function \"answer_uniswap_question\" *****\u001b[0m\n",
            " Uniswap v3 provides increased capital efficiency and fine-tuned control to liquidity providers, improves the accuracy and convenience of the price oracle, and has a more flexible fee structure. It also introduces multiple pools for each pair of tokens, each with a different swap fee, and introduces the concept of concentrated liquidity.\n",
            "\u001b[32m********************************************************************\u001b[0m\n",
            "\n",
            "--------------------------------------------------------------------------------\n",
            "\u001b[33massistant\u001b[0m (to user_proxy):\n",
            "\n",
            "Based on the answers, here is an introduction for your blog:\n",
            "\n",
            "\"Uniswap is a noncustodial automated market maker implemented for the Ethereum Virtual Machine. It has revolutionized the way we trade cryptocurrencies by providing a decentralized platform for swapping tokens. The latest version, Uniswap v3, brings a host of improvements and changes. It provides increased capital efficiency and fine-tuned control to liquidity providers, making it more beneficial for them to participate. The accuracy and convenience of the price oracle have been improved, providing more reliable price feeds. The fee structure has become more flexible, catering to a wider range of use cases. Uniswap v3 also introduces multiple pools for each pair of tokens, each with a different swap fee, and introduces the concept of concentrated liquidity. This allows liquidity providers to concentrate their capital within specific price ranges, increasing their potential returns.\"\n",
            "\n",
            "Now, let's convert this text to audio and speak it out loud.\n",
            "\u001b[32m***** Suggested function Call: convert_text_to_audio *****\u001b[0m\n",
            "Arguments: \n",
            "{\n",
            "  \"text\": \"Uniswap is a noncustodial automated market maker implemented for the Ethereum Virtual Machine. It has revolutionized the way we trade cryptocurrencies by providing a decentralized platform for swapping tokens. The latest version, Uniswap v3, brings a host of improvements and changes. It provides increased capital efficiency and fine-tuned control to liquidity providers, making it more beneficial for them to participate. The accuracy and convenience of the price oracle have been improved, providing more reliable price feeds. The fee structure has become more flexible, catering to a wider range of use cases. Uniswap v3 also introduces multiple pools for each pair of tokens, each with a different swap fee, and introduces the concept of concentrated liquidity. This allows liquidity providers to concentrate their capital within specific price ranges, increasing their potential returns.\"\n",
            "}\n",
            "\u001b[32m**********************************************************\u001b[0m\n",
            "\n",
            "--------------------------------------------------------------------------------\n",
            "\u001b[35m\n",
            ">>>>>>>> EXECUTING FUNCTION convert_text_to_audio...\u001b[0m\n",
            "First audio byte received in: 0.21162700653076172\n",
            "\u001b[33muser_proxy\u001b[0m (to assistant):\n",
            "\n",
            "\u001b[32m***** Response from calling function \"convert_text_to_audio\" *****\u001b[0m\n",
            "0\n",
            "\u001b[32m******************************************************************\u001b[0m\n",
            "\n",
            "--------------------------------------------------------------------------------\n",
            "\u001b[33massistant\u001b[0m (to user_proxy):\n",
            "\n",
            "TERMINATE\n",
            "\n",
            "--------------------------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "# the assistant receives a message from the user, which contains the task description\n",
        "user_proxy.initiate_chat(\n",
        "    assistant,\n",
        "    message=\"\"\"\n",
        "I'm writing a blog to introduce the version 3 of Uniswap protocol. \n",
        "Find the answers to the 2 questions below, write an introduction based on them and speak it out loudly.\n",
        "\n",
        "1. What is Uniswap?\n",
        "2. What are the main changes in Uniswap version 3?\n",
        "\n",
        "Start the work now.\n",
        "\"\"\"\n",
        ")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    },
    "vscode": {
      "interpreter": {
        "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
